Papers with training models

47 papers
Efficient Semi-supervised Consistency Training for Natural Language Understanding (2022.naacl-industry)

Copied to clipboard

Challenge: Manually labeled training data is expensive, noisy, and often scarce . semi-supervised learning methods can be used to improve model performance .
Approach: They explore different methods for consistency training on unlabeled data . they use human paraphrasing, back-translation, and dropout to augment unlabed data.
Outcome: The proposed methods outperform purely supervised learning on unlabeled data.
On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization (2024.eacl-short)

Copied to clipboard

Challenge: Text summarization and simplification are among the most widely used applications of NLP, but they are prone to hallucination due to training on unaligned data.
Approach: They propose a loss-truncation approach to modify the standard log loss to adaptively remove noisy examples during training to improve model performance.
Outcome: The proposed approach yields a considerable number of hallucinated entities on various datasets.
Improving Embedding-based Large-scale Retrieval via Label Enhancement (2021.findings-emnlp)

Copied to clipboard

Challenge: Existing methods for large-scale retrieval are trained with 0-1 hard labels that indicate whether a query is relevant to a document, ignoring rich information of the relevance degree.
Approach: They propose to introduce label enhancement for the first time to characterize query-document relevance degree by embedding label distribution into contextual embeddables.
Outcome: The proposed method significantly outperforms existing retrieval models and its counterparts equipped with two alternative methods on English and Chinese large-scale retrieval tasks.
DOCMASTER: A Unified Platform for Annotation, Training, & Inference in Document Question-Answering (2024.naacl-demo)

Copied to clipboard

Challenge: DOCMASTER is a platform for annotating PDF documents, model training, and inference, tailored to document question-answering.
Approach: They propose to integrate layout information into a unified platform for annotating PDF documents, model training, and inference tailored to document question-answering.
Outcome: The proposed platform is designed for annotating PDF documents, model training, and inference, tailored to document question-answering.
MWE as WSD: Solving Multiword Expression Identification with Word Sense Disambiguation (2023.findings-emnlp)

Copied to clipboard

Challenge: Recent approaches to word sense disambiguation use encodings of the sense gloss and context information to improve performance.
Approach: They propose a poly-encoder architecture which uses the sense gloss to improve WSD performance.
Outcome: The proposed approach outperforms the state-of-the-art in word sense disambiguation by 1.9 F1 points and on the PARSEME 1.1 English dataset.
LinggleWrite: a Coaching System for Essay Writing (2020.acl-demos)

Copied to clipboard

Challenge: Existing writing services that provide feedback on writing skills are not providing sufficient "coaching" information.
Approach: They propose a writing coach that provides writing suggestions, assesses writing proficiency levels, detects grammatical errors, and offers corrective feedback in response to user’s essay.
Outcome: The proposed system improves on public test sets and shows that both AES and GED models achieve state-of-the-art performance.
Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions (2023.findings-eacl)

Copied to clipboard

Challenge: a general-domain model has access to customer or domain specific parallel data at inference time, but not during training.
Approach: They propose a zero-shot adaptation approach where a general-domain model has access to customer or domain specific parallel data at inference time, but not during training.
Outcome: The proposed architecture outperforms existing architectures in two language pairs . it consistently improves BLEU across language pair, domain, and number k of fuzzy matches .
Multi-Source Text Classification for Multilingual Sentence Encoder with Machine Translation (2024.naacl-srw)

Copied to clipboard

Challenge: Pre-trained multilingual sentence encoders suffer from performance degradation for non-English languages.
Approach: They propose a method of machine translating a source sentence into English and then inputting it together with the source sentence in a multi-source manner.
Outcome: The proposed method improves the performance of pre-trained multilingual sentence encoders in Japanese on sentiment analysis and topic classification tasks.
Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method (2025.acl-long)

Copied to clipboard

Challenge: Existing long-context training data is scarce and requires substantial GPU resources for training.
Approach: They propose a training-free plug-and-play method to enhance long-context understanding in existing large language models.
Outcome: The proposed method outperforms existing LLMs on various tasks and surpasses baseline methods.
Multimodal Generation with Consistency Transferring (2025.findings-naacl)

Copied to clipboard

Challenge: Existing methods for multimodal content generation are limited to unimodal content production due to high training complexity, significant costs, and inadequate emphasis on model constraints.
Approach: They propose a method to generate multimodal content with constraints on adjacent steps and a layer-based layer-constrained transfer between adjacent steps to improve denoising capabilities.
Outcome: The proposed method improves the model’s ability to capture actions and depict backgrounds more effectively and improves video generation speed by approximately 40% and quality by about 39.3%.
Plasticity vs. Rigidity: The Impact of Low-Rank Adapters on Reasoning on a Micro-Budget (2026.eacl-srw)

Copied to clipboard

Challenge: Recent advances in mathematical reasoning typically rely on massive scale . yet, can strong reasoning capabilities be induced in small language models under extreme constraints?
Approach: They train small language models with a single GPU for under 24 hours . they find that adapters unlock significant plasticity in standard instruction-tuned models .
Outcome: The proposed model training on a single GPU (48GB) achieves 40% Pass@1 on AIME 24 (an 11.1% improvement over baseline) the model training results show that the adapter capacity and initialization are critical factors.
Learn from Failure: Fine-tuning LLMs with Trial-and-Error Data for Intuitionistic Propositional Logic Proving (2024.acl-long)

Copied to clipboard

Challenge: Recent advances in Automated Theorem Proving have shown the effectiveness of leveraging a (large) language model that generates tactics (i.e. proof steps) to search through proof states.
Approach: They propose to use a large language model that generates tactics to search through proof states.
Outcome: The proposed model solves more unseen theorems with lower trial searches than the current model, which only learns from failed attempts.
Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models (2025.acl-long)

Copied to clipboard

Challenge: Large language models (LLMs) have prioritized expanding the context window from which they can incorporate more information.
Approach: They propose a data augmentation strategy to enable large language models to gain long-context capabilities without the need to modify existing data mixture.
Outcome: The proposed model outperforms existing models on 20 billion tokens and achieves 75% and 84.5% accuracy on RULER at 128K context length.
Accurate Training of Web-based Question Answering Systems with Feedback from Ranked Users (2023.acl-industry)

Copied to clipboard

Challenge: Recent work shows that large-scale annotated datasets are essential for training state-of-the-art Question Answering (QA) models.
Approach: They use large-scale annotated datasets to train question answering models . they use feedback data collected from deployed QA systems to provide cheaper supervision .
Outcome: The proposed model improves on the large scale annotated datasets from QA systems . the proposed model can be easily supervised on large-scale unlabeled web data .
Little Giants: Synthesizing High-Quality Embedding Data at Scale (2025.naacl-long)

Copied to clipboard

Challenge: Synthetic data generation is an increasingly popular way of training models without the need for large, manually labeled datasets.
Approach: They propose a framework that aligns open-source small models to efficiently generate large-scale embedding data.
Outcome: The proposed framework outperforms state-of-the-art embedding models by using only 1/10 of the GPT API calls.
The Role of Data Curation in Image Captioning (2024.eacl-long)

Copied to clipboard

Challenge: Existing image captioning models treat all samples equally, neglecting mismatched data . Several other techniques have relied on curriculum learning strategies to adapt learning to the difficulty of the task.
Approach: They propose to actively curate difficult samples in datasets using curriculum learning strategies to improve captioning models.
Outcome: The proposed methods outperform existing models on the Flickr30K and COCO datasets.
Exploring Enhanced Code-Switched Noising for Pretraining in Neural Machine Translation (2023.findings-eacl)

Copied to clipboard

Challenge: Multilingual pretraining approaches to denoise synthetic code-switched data have shown that they generate the noise using non-contextual, one-to-one word translations obtained from lexicons.
Approach: They propose an approach where contextual, many-to-many word translations are generated using a ‘base’ NMT model.
Outcome: The proposed approach improves on 3 different language families and shows that small models can perform better than massive models like mBART50 and mRASP2 .
Certified Robustness to Programmable Transformations in LSTMs (2021.emnlp-main)

Copied to clipboard

Challenge: Existing techniques for certifying robustness of LSTMs and extensions of lsts are prone to adversarial examples.
Approach: They propose an approach to certify robustness of LSTMs and extensions of lstms . they show that their approach can train models more robust to combinations of string transformations - a key advantage of existing certification approaches .
Outcome: The proposed approach can show high certification accuracy of the resulting models.
CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideation (2026.acl-long)

Copied to clipboard

Challenge: a hallmark of human innovation is recombination.
Approach: They propose a task to extract recombination instances from scientific literature . they analyze patterns of recombined concepts and apply it to a broad corpus of AI papers .
Outcome: The proposed model can predict cross-disciplinary research directions . it can predict recombinations across areas and link methods and concepts .
CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation (2023.emnlp-main)

Copied to clipboard

Challenge: Annotated data plays a critical role in training models and evaluating their performance.
Approach: They propose a paradigm for Human-LLM co-annotation of unstructured texts at scale that utilizes uncertainty to estimate LLMs’ annotation capability.
Outcome: The proposed model outperforms existing models on many text-annotation tasks with up to 21% performance improvement over random baseline.
A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism (2024.findings-acl)

Copied to clipboard

Challenge: Multi-way parallel, machine generated content dominates the translations in lower resource languages . a limited investigation suggests this selection bias is the result of low quality content generated in English and translated into many lower resource language via MT.
Approach: They show that multi-way parallel, machine generated content dominates translations in many languages . they also find evidence of a selection bias in the type of content which is translated into many languages.
Outcome: The results suggest that the low quality of multi-way translations on the web was likely created using machine translation.
Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU (2022.emnlp-main)

Copied to clipboard

Challenge: Current approaches for NLU use CL to improve in-distribution data performance via heuristic-oriented or task-agnostic difficulties.
Approach: They propose to use CL to improve in-distribution data performance by taking advantage of training dynamics as difficulty metrics instead of heuristic-oriented or task-agnostic difficulties.
Outcome: The proposed model schedulers improve on in-distribution, out-of-distortion and zero-shot cross-lingual transfer datasets while being 20% faster on average.
Empowering Character-level Text Infilling by Eliminating Sub-Tokens (2024.acl-long)

Copied to clipboard

Challenge: Existing methods for character-level infilling relied on predicting sub-tokens, but this strategy was ineffective.
Approach: They propose a method to fill-in-the-mid with Starting and Ending character constraints that avoids predicting sub-tokens in inference.
Outcome: The proposed method surpasses existing methods and offers significant performance gains.
DynaSent: A Dynamic Benchmark for Sentiment Analysis (2021.acl-long)

Copied to clipboard

Challenge: Sentiment analysis is an early success story for NLP, in both a technical and an industrial sense.
Approach: They propose to combine naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation.
Outcome: The proposed model is more coherent than comparable models and motivates training models from scratch over successive fine-tuning.
ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws (2024.emnlp-main)

Copied to clipboard

Challenge: Existing quality filtering methods rely on a high-quality dataset as reference . Existing methods introduce potential biases and compromise diversity .
Approach: They propose a method that evaluates text quality based on the perplexity difference between two language models trained on the same data.
Outcome: The proposed approach improves performance of pre-trained models without increasing training costs.
Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity (2022.findings-emnlp)

Copied to clipboard

Challenge: Using low-resource languages, we assess the quality of uncertainty estimates from a wide array of approaches, but with more data.
Approach: They train models on sub-sampled datasets in three different languages to assess the confidence of a neural classifier.
Outcome: The proposed models train on sub-sampled datasets in three different languages and show that the quality of uncertainty estimates suffers with more data.
Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning (2021.naacl-main)

Copied to clipboard

Challenge: Prior work shows that disagreement between annotators can be useful in training models.
Approach: They propose to use disagreements as an auxiliary task in a multi-task neural network to incorporate disagreements into models.
Outcome: The proposed method significantly improves performance on NLP tasks beyond the standard approach and prior work.
Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning (2024.naacl-long)

Copied to clipboard

Challenge: Recent advances of Large Language Models (LLMs) have been pushing the field of Natural Language Processing (NLP) to the next level in many different aspects.
Approach: They propose a novel labeling method which estimates how much incremental knowledge is brought into LLMs by a demonstration.
Outcome: The proposed method estimates how much incremental knowledge is brought into the LLMs by a demonstration.
The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case Study (2022.acl-long)

Copied to clipboard

Challenge: Obtaining human-like performance in NLP is often argued to require compositional generalisation.
Approach: They re-instantiate three compositionality tests from the literature and reformulate them for neural machine translation.
Outcome: The proposed models are more compositional than models trained on more data, the authors show . they also show that some non-compositional behaviours are mistakes, whereas others reflect natural variation in data.
A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering (2021.acl-long)

Copied to clipboard

Challenge: Weakly supervised question answering usually has only final answers as supervision signals while correct solutions are not provided.
Approach: They propose to explicitly exploit the semantic correlations between question-answer pairs and predicted answers by maximizing mutual information between question and answer pairs.
Outcome: The proposed method significantly outperforms previous learning methods in terms of task performance and is more effective in training models to produce correct solutions.
Zero-Shot Cross-Lingual Transfer with Meta Learning (2020.emnlp-main)

Copied to clipboard

Challenge: There are more than 7,000 languages spoken in the world, over 90 of which have more than 10 million native speakers each.
Approach: They propose to use meta-learning to train a model on multiple languages at the same time . they use standard supervised, zero-shot cross-lingual, and few-shot crosses-lingual settings for different natural language understanding tasks.
Outcome: The proposed setup improves on the state-of-the-art for a total of 15 languages.
Code-Switching Curriculum Learning for Multilingual Transfer in LLMs (2025.findings-acl)

Copied to clipboard

Challenge: Large language models (LLMs) exhibit near human-level performance in various tasks, but performance drops after a handful of high-resource languages due to the imbalance in pre-training data.
Approach: They propose a code-switching curriculum learning model to enhance cross-lingual transfer for LLMs by progressively training models with a curriculum consisting of token-level code-changing, sentence-level codeswitching, and monolingual corpora.
Outcome: The proposed model improves language transfer to Korean, with significant gains in Japanese and Indonesian . the proposed model mitigates spurious correlations between language resources and safety alignment .
Distillation with Explanations from Large Language Models (2024.lrec-main)

Copied to clipboard

Challenge: Large language models (LLMs) like ChatGPT and GPT-4 have made remarkable progress in various NLP tasks while also providing explanations alongside their answers.
Approach: They propose to use Large language models (LLMs) to generate more accurate answers and corresponding free-text explanations by combining ground truth labels and answers-explanations generated by LLMs.
Outcome: The proposed method achieves improved predictive performance and generates explanations that exhibit greater alignment with the model’s task outputs.
Does Self-Rationalization Improve Robustness to Spurious Correlations? (2022.emnlp-main)

Copied to clipboard

Challenge: Rationalization is fundamental to human reasoning and learning.
Approach: They evaluate robustness to spurious correlations in encoder-decoder and decoder-only models . authors say explanations can come at the cost of robustness .
Outcome: The proposed model outputs are more interpretable and easier to interact with for end-users than nonrationalizing models.
Revisiting Distant Supervision for Relation Extraction (L18-1)

Copied to clipboard

Challenge: Existing approaches for relation extraction (RE) use supervised learning on relation-specific training data, which is expensive to acquire.
Approach: They propose to use a new testing dataset to re-examine distant supervision approaches . they aim to draw new conclusions based on the new testing data .
Outcome: The proposed method can generate training data without noise and bias issues . the proposed method is annotated by the researchers on Amzaon Mechanical Turk .
Leveraging Synthetic Targets for Machine Translation (2023.findings-acl)

Copied to clipboard

Challenge: Using synthetic target data, training models on synthetic targets outperforms training on actual ground-truth data.
Approach: They propose a recipe for training machine translation models on synthetic target data by leveraging a large pre-trained model.
Outcome: The proposed model outperforms training on real-world translation datasets.
How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark (2025.emnlp-main)

Copied to clipboard

Challenge: Prior work has not explored the mechanisms underlying this sensitivity.
Approach: They propose a synthetic benchmark to evaluate Large Language Models’ reasoning robustness against systematically controlled irrelevant context (IC).
Outcome: The proposed model improves in-distribution and out-of-disttribution scenarios while training with strong distractors.
Weaker Than You Think: A Critical Look at Weakly Supervised Learning (2023.acl-long)

Copied to clipboard

Challenge: Weakly supervised learning is a popular approach for training machine learning models in low-resource settings.
Approach: They propose to use weakly supervised learning to train models with noisy labels from weak sources instead of collecting expensive human annotations.
Outcome: The proposed methods outperform weakly supervised methods on various NLP datasets and tasks on the test sets.
Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments (2025.acl-long)

Copied to clipboard

Challenge: Specialized non-LLM NLP-based solutions lack reasoning and are not able to infer values not explicitly present in documents.
Approach: They propose a novel LLM-based approach that organizes VRDs into localized semantic textual segments called semantic blocks.
Outcome: The proposed approach outperforms the state-of-the-art on public VRD benchmarks by 1-3% in F1 scores and is resilient to document formats previously not encountered.
Structured Preference Optimization for Vision-Language Long-Horizon Task Planning (2025.emnlp-main)

Copied to clipboard

Challenge: Existing vision-language planning methods struggle with long-horizon reasoning in dynamic environments due to the difficulty of training models to generate high-quality reasoning processes.
Approach: They propose a framework that enhances reasoning and action selection for long-horizon task planning through structured evaluation and optimized training.
Outcome: The proposed framework outperforms existing methods on short-horizon tasks but struggles with long-horizon reasoning in dynamic environments.
Don’t Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models (2026.acl-long)

Copied to clipboard

Challenge: Small language models struggle with tool-use tasks, particularly in selecting appropriate tools and identifying correct parameters.
Approach: They propose a training-free method that leverages peakedness to align schemas with pretraining knowledge to rename tool components.
Outcome: Experiments on MetaTool and RoTBench show that PA-Tool significantly improves tool-use accuracy without retraining.
USDC: A Dataset of  ̲User  ̲Stance and  ̲Dogmatism in Long  ̲Conversations (2025.findings-acl)

Copied to clipboard

Challenge: Previously, studies on stance and dogmatism in user conversations have focused on training models using annotated datasets at the post level, treating each post as independent and randomly sampling posts from conversation threads.
Approach: They build a dataset for studying user opinion fluctuations in 764 long multi-user Reddit conversation threads, called USDC.
Outcome: The proposed dataset analyzes user opinion fluctuations in 764 long multi-user Reddit conversation threads.
From Implicit to Explicit: Token-Efficient Logical Supervision for Mathematical Reasoning in LLMs (2026.findings-acl)

Copied to clipboard

Challenge: Recent studies reveal large language models lack logical reasoning abilities . logical relationship understanding is a core capability underlying genuine logical thinking .
Approach: They propose a lightweight training framework targeting logical relationship understanding . they propose logical relation understanding framework that provides explicit supervision .
Outcome: The proposed framework outperforms CoT-SFT training frameworks in logical relationship understanding tasks.
HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router (2026.acl-long)

Copied to clipboard

Challenge: Current alignment approaches rely on refusal alignment to avoid harmful content . large language models are often overly cautious or overlook subtle harmful content.
Approach: They propose a framework for fine-grained safe generation in Large Language Models that enables real-time, token-level harmfulness detection and redaction without loss in capability.
Outcome: The proposed framework achieves over 90% in F1 score for detecting and redacting harmful content while preserving overall utility and informativeness of the model’s responses.
Incomplete Prompt Jailbreaks in Large Language Models (2026.findings-acl)

Copied to clipboard

Challenge: Large language models (LLMs) are increasingly released as open-weight models with safeguards against harmful requests.
Approach: They formalize incomplete prompt jailbreaks as incomplete prompts elicit harmful continuations . they identify two functional neurons that delay refusal until sentence termination .
Outcome: The proposed model fails to generalize across content domains and attractor types . the proposed model can be used to perform more precise and robust IPJ defenses .
J-Shuwa: A Large-Scale Web-Collected Japanese Sign Language-Japanese Parallel Corpus (2026.findings-acl)

Copied to clipboard

Challenge: Japanese Sign Language (JSL) is a low-resource sign language that has received limited attention in the AI community due to the lack of large-scale, publicly available parallel corpora.
Approach: They propose a large-scale JSL-Japanese parallel corpus constructed from YouTube videos with hard-coded subtitles and closed captions.
Outcome: The proposed model is effective for training models and can be used for future research across a wide range of tasks.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations